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Operationalizing  Halo:  Problems  with  the  Computation 
of  a  Standard  Deviation  Across  Dimensions 
Within  Ratees 

Conceptual  discussions  of  halo  have  been  relatively  free 
of  inconsistencies.  For  example ,  halo  has  been  defined  as  a 
tendency  to  attend  to  a  global  impression  of  each  ratee  rather 
than  to  carefully  distinguish  among  levels  of  different  performance 
dimensions  (Borman,  1975);  a  rater's  inability  or  unwillingness 
to  distinguish  among  the  dimensions  of  a  ratee 's  job  behavior 
(DeCotiis,  1977);  and  a  tendency  to  place  a  given  ratee  at  the 
same  level  on  different  dimensions  (Beraardln,  1977).  Thus,  the 
halo  effect  is  generally  considered  as  a  rater's  failure  to 
discriminate  among  conceptually  distinct  and  possibly  independent 
aspects  of  a  ratee 's  performance  which,  in  turn,  results  in 
higher  dimension  intercorrelations  than  the  "true"  level  of  these 
intercorrelations . 

« 

Although  there  is  substantial  agreement  concerning  the 
conceptualization  of  halo,  there  is  little  consensus  concerning 
how  it  should  be  measured.  For  instance,  one  approach  is  to 
examine  the  interdimenaion  factor  structure.  To  the  degree  that 
this  structure  is  dominated  by  a  general  factor  accounting  for 
an  appreciable  portion  of  the  rating  variance,  halo  is; thought  to 
be  present  (Kraut,  1975).  A  second  approach  is  baaed  on  a  Rater 
x  Ratee  x  Dimension  analysis  of  variance  (Guilford,  1954;  Kavanagh, 


Page-4 


Pulakos  &  Schmitt 


-I 


MacKinney,  &  Wollins,  1971),  in  which  a  statistically  significant 
Rater  x  Ratee  interaction  (especially  one  that  explains  a  large 
proportion  of  the  variance)  is  indicative  of  halo.  Some  authors 
have  suggested,  however,  that  this  method  is  somewhat  of  an 
oversimplification  (Stanley,  1961;  Villlngham  &  Jonea,  1958). 

A  third  approach  is  to  calculate  the  inter  dimension  correlations 
and  to  draw  inferences  about  whether  or  not  these  intercorrelations 
are  higher  than  whet  is  thought  to  be  their  "true"  value  (Thorndike, 
1920).  A  fourth,  e  perhaps  the  most  common,  way  of  measuring 
halo  is  to  calculate  the  standard  deviation  (SD)  associated  with  a 
given  rater's  ratings  of  a  particular  ratee  across  ell  performance 
dimensions  (Berner din  g  Pence,  1979;  Beraardln  &  Walter,  1977, 

Borman,  1975). 

A  major  problem  surrounding  these  operationalizations  of 
halo  is  that  none  of  then  considers  nor  csn  they  consider  the 
degree  to  which  the  rating  dimensions  ere  actually  correlated. 

Hence,  the  adequacy  of  these  measures  for  assessing  valid  versus 
invalid  halo  is-  suspect.  Further,  as  Seal,  Downey,  and  Lahey  (1980) 
and  Cooper  (1981)  have  noted,  these  measures  are  neither  conceptually 
nor  empirically  equivalsnt.  Regarding  the  latter  two  approaches, 
for  example,  the  SD  method  measures  the  degree  to  which  ratings 
are  the  same  across  the  dimensions,  such  that  those  which  contain 
complete  halo  have  a  variance  of  zero  within  ratees.  Alternatively, 
the  correlational  approach  equates  halo  with  dimension 
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intercorrelations  equal  to  1.00.  There  is  also  a  significant 
problem  with  the  SD  measure  of  halo  in  that  it  will  be  nonzero 
simply  as  a  function  of  actual  mean  differences  across  the  rating 
dimensions.  However,  as  is  shorn  here,  use  of  the  SO  criterion 
with  data  that  have  been  standardised  within  rating  categories 
such  that  the  dimension  means  and  standard  deviations  are 
equivalent  corrects  this  oversight. 

The  purpose  of  the  present  paper  was  to  examine  differences 
between  computations  of  standardized  and  unstandardized  SB  criterla  - 
relative  to  a  third  measure  of  halo  which  considered  the  true  level 
of  Intercorrelation  among  the  dimensions.  Specifically,  the  SD  and 
standardized  SD  measures  were  correlated  with  the  average  difference 
between  the  true  and  observed  dimension  Intercorrelations  (an 
operationalization  of  halo  more  directly  consistent  with  conceptual 
discussions).  It  is  shown  by  example  that  standardizing  the  scores 
within  dimensions  prior  to  computing  the  SD  measure  across  dimensions 
for  each  rates  not  only  takes  into  account  irrelevant  naan 
differences  among  rating  categorise,  but  it  also  yields  a  halo 
measure  that  is  perfectly  correlated  with  the  difference  between  the 
"true"  and  observed  dimension  inter correlations  for  a  given  r*tar. 

Method 

Subjects  - - 

One  hundred  and  eight  undergraduate  students  enrolled  in  an 
industrial/organizational  psychology  course  participated  in  the 
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study.  The  total  sample  consisted  of  58  males  and  50  females, 
whose  mean  age  was  20.64  years. 

Rating  Task 

Subjects  viewed  5-  to  9-oinute  videotapes  of  six  managers 
talking  with  a  problem  subordinate.  Ratings  of  each  manager's 
performance  were  made  using  five  behaviorally-based  rating 
scales  representing  the  following  dimensions  of  the  manager's 
job: 

1.  Structuring  and  Controlling  the  Interview 

2.  Establishing  and  Maintaining  Rapport 

3.  Resolving  Conflict 

4.  Motivating  the  Subordinate 

5.  ’  Developing  the  Subordinate 

Each  dlmanaion  waa  defined  by  an  overall  defining  statement 
as  well  as  by  seven,  scaled  behavioral  anchors  describing 
different  effectiveness  levels. 

Videotaped  performances  were  used  because  they  enabled 
the  calculation  of  "true1'  performance  for  each  rates  and  hence 
the  true  levels  of  Intercorrelation  bebwen  the  rating  dimensions. 
The  videotapes  used  here  were  carefully  developed  so  as  to 
insure  that  the  performances  represented  a  variety  of  effectiveness 
levels  on  different  rating  dimensions.  Specific  details  regarding 
the  development  of  the  tapes,  the  rating  scales,  and  the  procedure 
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used  to  generate  true  scores  of  performance  for  each  manager  can 
be  found  in  Borman  (1977). 

Halo  Measures 

Standard  Deviations  (measure  often  used).  Operationally , 
halo  has  been  discussed  in  terms  of  standard  deviations  across 
dimensions  within  ratees  (e.  g. ,  Borman,  197S).  A  standard  deviation 
was  thus  computed  for  each  target  ratee,  reflecting  the  spread  in 
their  ratings  across  the  dimensions.  Subjects'  standard  deviations 
for  each  of  the  six  ratees  were  then  averaged  to  provide  the  final 
halo  measure.  In  previous  studies,  a  low  standard  deviation  across 
dimensions  has  been  indicative  of  more  halo,  while  a  high  standard 
deviation  has  bean  Indicative  of  lass  halo. 

Standardised  Standard  Deviations.  For  the  five  performance _ _ 

dimensions,  ratings  ware  standardised  across  ratees,  resulting  in 
dimension  means  of  sero  and  standard  deviations  of  one.  Standard 
deviations  ware  'then  calculated  across  the  five  standardized 
dimensions  for  each  rates.  Finally,  these  sis  (each  rater  viewed  six 
videotaped  interviews)  standard  deviations  ware  averaged  to  provide 
the  final  measure  of  the  degree  to  which  each  rater's  ratings 
contained  halo  error  as  defined  above. 

Dimension  Intercorrelatlona.  The  third  measure  of  halo  ~ 
was  calculated  by  computing  a  correlation  matrix  between  the  five 
dimensions  for  each  subject's  ratings  of  the  six  ratees.  These 
dimension  intercorrelations  were  then  subtracted  from  the  true 
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dimension  intercorrelations ,  yielding  10  difference  scores  for 
each  subject.  Prior  to  subtracting  the  matrices,  all  correlations 
were  transformed  to  z  scores  using  Fisher’s  r-to-z  transformation. 

The  difference  scores  were  then  averaged,  providing  a  mean  measure 
of  the  difference  between  the  true  and  observed  intercorrelations 
across  dimensions.  To  the  degree  that  this  average  deviated  from  zero 
in  a  positive  direction,  the  subject's  ratings  were  less  correlated 
than  the  true  ratings.  To  the  degree  that  this  average  deviated 
from  zero  in  a  negative  direction,  greater  halo  was  evidenced. 

Reaults  and  Discussion 

Presented  in  Table  1  are  the  correlations  between  the  three 
halo  measures  described  above.  As  can  be  seen  from  this  table, 
the  standardized  standard  deviation  and  the  difference  between 
true  and  observed  intercorrelation  measures  of  halo  are  nearly 
perfectly  correlated.  The  absence  of  a  1.00  correlation  between 
these  measures  is  likely  due  to  rounding  error.  Further,  the 
relationship  between  these  two  measures  and  the  standard  deviation 
operationalization  of  halo  is  leas  (approximately  .80). 


Insert  Table  1  about  here 


Inspection  of  the  rating  dimension  means  revealed  the 
following:  Structuring  and  Controlling  the  Interview  (x  •  A. 16), 
Ear  blishi'  and  Maintaining  Rapport  (x  ■  4.38),  Resolving  Conflict 
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(x  -  3.65),  Motivating  the  Subordinate  (x  ■  4.12),  and  Developing 
the  Subordinate  (x  •  3.98).  Although  there  was  some  variation  in 
these  means,  extreme  differences  were  not  present.  However,  given 
relatively  equal  standard  deviations  within  the  dimensions,  larger 
■wan  differences  would  have  resulted  in  a  lower  correlation  between 
the  standardized  and  unstandardized  standard  deviation  measures  of 
halo.  Therefore,  the  differences  that  we  observed  were  likely  to 
be  small  compared  to  what  would  be  observed  in  many  studies. 

With  a  correlation  of  at  leaat  .80  between  the  halo  measures, 
the  practical  differences  associated  with  using  one  measure  versus 
another  may  not  seem  particularly  important.  However,  the  data 
reported  hare  were  collected  as  part  of  a  training  study  in  which 
a  significant  main  effect  for  accuracy  training,  F  (1,  106)  •  7.06, 

£.  <  .05,  resulted  for  the  SD  (average  standard  deviation  within 
ratees)  measure  of  halo,  but  a  nonsignificant  main  effect,  F  (1,  106) 

•  .08,  ns.,  resulted  for  the  sverage  difference  between  true  and 
observed  dimension  intercorrelations  measure  of  halo  (Pulakos,  1983). 

The  arguments  and  data  presented  here  suggest  that  measuring 
halo  by  calculating  a  standard  deviation  within  ratees  is  not  entirely 
appropriate.  This  operationalization  neither  takes  into  account 
irrelevant  dimension  mean  differences  nor  is  it  entirely  consistent  * 

with  conceptual  discussions  of  halo  (i.  e. ,  higher  observed  dimension 
intercorrelations  than  the  "true"  levels  of  these  intercorrelations). 

However,  by  standardizing  a  rater's  ratings  within  each  dimension 

i 
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prior  to  calculating  standard  deviations  across  the  diaensions  for 
each  ratee,  a  measure  of  halo  results  that  is  equivalent  to  (i.  e., 
perfectly  correlated  with)  the  average  difference  between  true  and 
observed  dimension  intercorrelations.  This  latter  measure  is  also 
perfectly  correlated  with  the  absolute  level  of  intercorrelation 
of  the  dimensions  for  a  given  rater  (the  relationship  between  the 
average  absolute  level  of  intercorrelation  and  the  average  difference 
between  the  true  and  observed  dimension  inter correlations  in  the 
present  data  was  r,  -  1.00).  This,  of  course,  is  a  result  of  the 
fact  that  subtracting  constants  (i.  e.,  the  true  intercorrelations) 
does  not  affect  the  nature  of  the  relationship  itself. 

In  conclusion,  than,  equivalent  measures  of  halo  are  obtained 
by  using  any  of  three  operationalisations  of  the  error:  1)  by 
computing  the  average  standard  deviation  for  each  rater  across 
standardised  within  dimension  scores;  2)  by  calculating  a  rater's 
average  level  of  observed  dimension  intercorrelation;  or  3)  by 
calculating  the  average  difference  between  the  true  and  observed 
dimension  intercorrelations  for  a  given  rater.  Further,  although 
the  unstandardized  SD  measure  of  halo  may  be  substantially  correlated 
with  theae  measures,  practically  important  differences  can  result 
(a a  shown  here)  in  statistical  analyses  using  this  operationalization 
versus  one  of  the  other  three.  Thus,  the  frequently  used  SD  measure 
of  halo  is  not  recommended  for  future  assessments  of  the  error. 

Although  computing  the  SD  across  standardized  diaensions  is 
a  better  operationalization  of  halo  than  computing  a  SD  across 
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unstandardized  dimensions,  a  cautionary  note  seems  warranted. 

That  is,  the  actual  level  of  halo  is  indeterminant  in  the  absence 
of  the  true  intercorrelations  among  dimensions.  Thus,  even  in 
using  a  standard  deviation  calculated  from  standardized  dimension 
scores,  it  is  still  not  known  whether  or  not  a  given  SO  (e.  g.,  .50) 
is  too  large  or  too  small.  Similarly,  if  a  measure  is  truly 
multidimensional,  an  average  intercorrelation  amoqg  dimensions 
of  1.00  is  obviously  too  high;  but,  without  knowledge  of  the 
true  intercorrelatlona  among  the  dimensions,  whether  or  not  an 
observed  intercorrelation  of  .80  or  .30,  for  example,  is  too 
large  or  small  is  equally  ambiguous.  Thus,  data  generated  with 
the  as  measures  can  only  bo  discussed  in  relative  tone  rather  then- 
in  tarns  of  the  level  of  invalid  versus  valid  halo  present  In  the 
ratings. 
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Table  1 

Correlations  Between  the  Halo  Measures 


Standard  Deviation  (SD) 


SD  SSD  DI 


Standardized  Standard 

Deviation  (SSD)  .81 

Observed-True  Dimension 

Intercorrelations  (DI)  -.80  -.99 
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LIST  4  NAVMAT  &  NPRDC 


Program  Administrator  for  Manpower 
Personnel,  and  Training 
MAT-0722 

800  N.  Quincy  Street 
Arlington,  VA  22217 

Naval  Material  Command 
Management  Training  Center 
NAVMAT  09M32 

Jefferson  Plata,  Bldg  #2,  Rm  150 
1421  Jefferson  Davis  Highway 
Arlington,  VA  20360 


Naval  Material -Command 

Director,  Productivity  Management  Office 

MAT-OOK 

Crystal  Plata  #5 
Room  632 

Washington,  D.C.  20360 

Naval  Personnel  RAD  Center  (4) 

Technical  Director 
Director,  Manpower  &  Personnel 
Laboratory,  Code  06 
Director,  System  Laboratory,  Code  07 
Director,  Future  Technology,  Code  41 
San  Diego,  CA  92152 


•Number  In  parentheses  is  the  number  of  copies  to  be  sent. 
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J 


Navy  Personnel  R&D  Center 
Washington  Liaison  Office 
Ballston  Tower  #3,  Room  93 
Arlington,  VA  22217 


LIST  5  BUMED 


NONE 


LIST  6 

NAVAL  ACADEMY  AND  NAVAL  POSTGRADUATE  SCHOOL 


Naval  Postgraduate  School  (3) 

ATTN:  Chairman,  Dept  of 
Administrative  Science 
Department  of  Administrative  Sciences 
Monterey,  CA  93940 


U.S.  Naval  Academy 
ATTN:  Chairman,  Department  of 
Leadership  and  Law 
Stop  7-B 

Annapolis,  MD  21402 


LIST  7  HRM 


Officer  in  Charge  Human  Resource  Management  School 

Hiaao  Resource  ManagesMnt  Division  Naval  Air  Station  Memphis  (96) 

Navel  Air  Station  Millington,  TN  38054 

Mayport,  FL  32228 

Commanding  Officer 
Human  Reaource  Management  School 
Naval  Air  Station  Mamphla 
Millington,  TN  38054 

LIST  8  NAVY  MISCELLANEOUS 

Naval  Military  Personnel  Commend  (2) 

HRM  Department  (NMPC-6) 

Washington,  D.C.  20350 


LIST  9  USMC 


Headquarters,  U.S.  Marine  Corps 
ATTN:  Scientific  Adviser, 

Code  RD-1 

Washington,  D.C.  20380 


LIST  10  OTHER  FEDERAL  GOVERNMENT 


Dr.  Brian  Usllaner 
GAO 

Washington,  D.C.  20548 

Office  of  Personnel  Management 
Office  of  Planning  and  Evaluation 
Research  Management  Division 
1900  E.  Street,  N.W. 

Washington,  D.C.  20415 


Social  and  Developmental  Psychology 
Program 

National  Science  Foundation 
Washington,  D.C.  20550 


LIST  11  ARMY 


Technical  Director  (3) 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

LIST  12  AIR 

Air  University  Library 
LSE  76-443 

Maxwell  AFB,  AL  36112 


Head,  Department  of  Behavior 

Science  and  Leadership 

U.S.  Military  Academy,  New  York  10996 


FORCE 

Head,  Department  of  Behavioral 

Science  and  Leadership 

U.S.  Air  Force  Academy ,  CO  80840 


LIST  13  MISCELLANEOUS 


Mr.  Luigi  Petrullo 

2431  North  Edgewood  Street 

Arlington,  VA  22207 

LIST  14  CURRENT  CONTRACTORS 


Dr.  Janet  L.  Barn es-Farr ell 
Department  of  Psychology 
University  of  Hawaii 
2430  Campus  Road 
Honolulu,  HI  96822 

Jeanne  M.  Brett 
northwestern  University 
Graduats  School  of  Management 
2001  Sheridan  Road 
Evanston,  IL  60201 

Dr.  Terry  Connolly 
Georgia  Institute  of  Technology 
School  of  Industrial  &  Systems 
Engineering 
Atlanta,  GA  30332 

Dr.  Richard  Daft 
Texas  AAM  University 
Department  of  Management 
College  Station,  TX  77843 

Dr.  Randy  Dunham 
University  of  Wisconsin 
Graduate  School  of  Business 
Madison,  VI  53706 


Dr.  Lawrence  R.  James 
School  of  Psychology 
Georgia  Institute  of  Technology 
Atlanta,  GA  30332 

Dr.  J.  Richard  Hackman 

School  of  Organisation  &  Management 

Box  1A,  Tale  University 

New  Haven,  CT  06520 

Dr.  Frank  J.  Landy 
The  Pennsylvania  State  University 
Department  of  Psychology 
417  Bruce  V.  Moore  Building 
University  Park,  PA  16802 

Dr.  Bibb  Latane 

The  University  of  North  Carolina 
at  Chapel  Hill 
Manning  Hall  026A 
Chapel  Hill,  NC  27514  - 

Dr.  Edvard  E.  Lawler 
University  of  Southern  California 
Graduate  School  of  Business  Administration 
Los  Angeles,  CA  90007 


Dr.  William  H.  Mobley 

College  of  Business  Administration 

Texas  A&M  University 

College  Station,  TX  77843 

Dr.  Thomas  M.  Ostrom 
The  Ohio  State  University 
Department  of  Psychology 
116E  Stadium 
404C  West  17th  Avenue 
Columbus,  OH  43210 

Dr.  Robert  Rice 

State  University  of  New  York  at  Buffalo 
Department  of  Psychology 
Buffalo,  NY  14226 

Dr.  Benjamin  Schneider 
Department  of  Psychology 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  H.  Wallace  Slnalko 
Program  Director,  Manpower  Research 
and  Advisory  Services 
Smithsonian  Institution 
801  N.  Pitt  Street,  Suite  120 
Alexandria,  VA  22314 

Dr.  Richard  M.  Steers 
Graduate  School  of  Management 
University  of  Oregon 
Eugene,  OR  97403 

Dr.  Harry  C.  Tr land is 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

Dr.  Anne  S.  Tsui 

Duke  University 

The  Fuqua  School  of  Business 

Durham,  NC  27706 

Andrew  H.  Van  de  Van 
University  of  Minnesota 
Office  of  Research  Administration 
1919  University  Avenue 
St.  Paul,  MS  5S104 


